{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1.模型构造"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用自定义类继承自`nn.Module`，需要自定义的包括：\n",
    "\n",
    "1. 在构造函数中定义需要用到的网络层\n",
    "2. 自定义`forward`函数，规定前相传播的计算过程（`nn.Sequential`的计算过程是严格按照其中包含的层序进行的）\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2.参数管理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1参数访问\n",
    "\n",
    "对于一个层对象中的参数，通过`state_dict()`访问，返回一个包含所有参数的dict\n",
    "\n",
    "如：`nn.Linear.state_dict()`\n",
    "\n",
    "`weight`和`bias`中又包含两个属性：`weight`和`grad`，分别为参数的具体值和梯度值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OrderedDict([('weight', tensor([[ 0.2137, -0.0377,  0.0331, -0.0497, -0.1235,  0.2806,  0.2334,  0.2007,\n",
      "         -0.2862, -0.0361]])), ('bias', tensor([0.1923]))])\n",
      "tensor([[ 0.2137, -0.0377,  0.0331, -0.0497, -0.1235,  0.2806,  0.2334,  0.2007,\n",
      "         -0.2862, -0.0361]])\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "\n",
    "class MyModule(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(MyModule, self).__init__()\n",
    "        self.linear = nn.Linear(10,1)\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.linear(x)\n",
    "\n",
    "    def param_traverse(self):\n",
    "        print(self.linear.state_dict())\n",
    "        print(self.linear.weight.data)\n",
    "        print(self.linear.weight.grad)\n",
    "\n",
    "myMod = MyModule()\n",
    "myMod.param_traverse()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2参数初始化"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对一个模块使用`apply`函数，`apply`函数的参数为调用该`apply`的`nn.Module`对象及其子模块\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.6841,  0.3849,  0.2867,  0.2755,  0.1669, -0.8730,  0.3985, -0.3791,\n",
      "          0.5592,  0.3887]])\n",
      "tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])\n"
     ]
    }
   ],
   "source": [
    "def xavier(layer):\n",
    "    if type(layer) == nn.Linear:\n",
    "        nn.init.xavier_normal_(layer.weight)\n",
    "\n",
    "def myInit(layer):\n",
    "    if type(layer) == nn.Linear:\n",
    "        # nn.init.constant_(layer.weight, 0)\n",
    "        layer.weight.data = torch.zeros_like(layer.weight.data)\n",
    "\n",
    "linear = nn.Linear(10,1)\n",
    "linear.apply(xavier)\n",
    "print(linear.weight.data)\n",
    "linear.apply(myInit)\n",
    "print(linear.weight.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "修改某些模块中的某几个数据："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[42.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]])\n"
     ]
    }
   ],
   "source": [
    "with torch.no_grad():\n",
    "    linear.weight.data[0, 0] = 42   # 第0行，第0列元素\n",
    "print(linear.weight.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.模型保存"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一般来说，pytorch无法把网络结构保存下来，只保存权重"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([1, 2, 3, 4])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\13566\\AppData\\Local\\Temp\\ipykernel_27512\\92113128.py:5: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  x_ = torch.load('./tmp/x-file')\n"
     ]
    }
   ],
   "source": [
    "x = torch.tensor([1,2,3,4])\n",
    "\n",
    "torch.save(x, './tmp/x-file')\n",
    "\n",
    "x_ = torch.load('./tmp/x-file')\n",
    "print(x_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\13566\\AppData\\Local\\Temp\\ipykernel_27512\\659766791.py:8: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  dic_1 = torch.load('./tmp/dict')\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'x': tensor([0, 1, 2, 3]), 'y': tensor([0., 0., 0., 0.])}"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dic_ = {\n",
    "    'x': torch.arange(4),\n",
    "    'y': torch.zeros(4)\n",
    "}\n",
    "\n",
    "torch.save(dic_, './tmp/dict')\n",
    "\n",
    "dic_1 = torch.load('./tmp/dict')\n",
    "dic_1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "假设定义一个MLP，将其各层的权重保存下来\n",
    "\n",
    "之前说过，模型的`state_dict()`函数返回一个包含模型每一层权重参数的字典，具体格式如下：\n",
    "\n",
    "```py\n",
    "OrderedDict([\n",
    "    ('layer1_name.weight', tensor(...)),\n",
    "    ('layer1_name.bias', tensor(...)),\n",
    "    ('layer2_name.weight', tensor(...))\n",
    "])\n",
    "```\n",
    "所以，模型权重的保存，可以直接调用该函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "\n",
    "class MLP(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(MLP, self).__init__()\n",
    "        self.linear = nn.Linear(20, 256)\n",
    "        self.relu = nn.ReLU()\n",
    "        self.output = nn.Linear(256, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.output(self.relu(self.linear(x)))\n",
    "    \n",
    "net = MLP()\n",
    "\n",
    "# 保存各层权重\n",
    "torch.save(net.state_dict(), './tmp/mlp.params')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "读取模型权重"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MLP(\n",
      "  (linear): Linear(in_features=20, out_features=256, bias=True)\n",
      "  (relu): ReLU()\n",
      "  (output): Linear(in_features=256, out_features=10, bias=True)\n",
      ")\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\13566\\AppData\\Local\\Temp\\ipykernel_27512\\2139904358.py:3: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  clone.load_state_dict(torch.load('./tmp/mlp.params'))\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([[True, True, True, True, True, True, True, True, True, True],\n",
       "        [True, True, True, True, True, True, True, True, True, True]])"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clone = MLP()\n",
    "\n",
    "clone.load_state_dict(torch.load('./tmp/mlp.params'))\n",
    "\n",
    "# 查看模型结构\n",
    "print(clone.eval())\n",
    "\n",
    "# 给两个网络输入同一个数据，看结果是否相同\n",
    "x = torch.randn(size=(2, 20))\n",
    "out_net = net.forward(x)\n",
    "out_clone = clone.forward(x)\n",
    "out_clone == out_net"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "torch",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
